Training motion with observation-based feedback

ABSTRACT

A system and method for training a student to perform bodily movements are disclosed. A model example of a bodily motion is communicated to the student, typically via video display of a model video example of the bodily motion. The student practices the bodily motion, during which an electronic observation system observes the practice. Based on such observation, feedback is provided to the user, which may include affecting: (i) subsequent video displays relating to the bodily motions; (ii) sessions in which sensory stimulus perceivable by the user is suppressed to facilitate training; and (iii) practice sessions in which the student practices the bodily motion.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of and priority to U.S.Provisional Application No. 61/544,495 filed Oct. 7, 2011 the content ofwhich is incorporated herein by reference for all purposes.

BACKGROUND

A variety of methods exist for training individuals to perform bodilymotions. Individuals often will watch bodily motions being performed byanother, for example an instructor or another expert at the motion. Thetrainee then attempts to replication that motion. In some cases, theirattempts are recorded using video or other means. In any case, thetrainee's performance is reviewed to assess whether and to what extentit varied from the motion as demonstrated (e.g., by the instructor).This assessment is used to guide improvement in successive practiceattempts.

The above method is often quite effective at teaching the bodily motion,but it is time-consuming and there is always a desire that theimprovements be greater and occur sooner. One reason for the limitationis simply that it takes a lot of time and practice for the trainee totruly “feel” the motion with their body in a way that allows “musclememory” to effectively take over and produce an optimal result. Also,the nature of the feedback, which often is in the form of verbalinstructions or additional demonstrations, is limited in its capacity torapidly produce significant improvements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary system for training bodily motions.

FIG. 2 depicts an exemplary method for training bodily motions.

FIGS. 3-6 depict model video examples of a bodily motion, along withrecordings of a student's attempt to perform the bodily motion.

FIG. 7 depicts an exemplary method for training bodily motions usingstimulation applied to various body parts.

FIG. 8 depicts a student engaged in body-stimulation motion training

DETAILED DESCRIPTION

FIGS. 1 and 2 show a system 100 and method 200 for training individualsto perform bodily movements. Such individuals may include athletes,physical therapy or other medical patients, participants in yogaclasses, individuals wanting to improve posture, workplace training,flexibility or range of motion, to name a few non-limiting examples.These individuals may work with coaches, doctors, physical oroccupational therapists, or others that may help the individual in theirmotion learning and practice. The two individuals in this trainingendeavor will be referred to in general as the “teacher” and “student,”unless the context lends itself to a more specific identification, suchas “coach” and “athlete.”

FIG. 1 shows a teacher T and student S that may interact in various wayswith the student. System 100 may be implemented in whole or in part as acomputing device having software and hardware components such as thosefound in desktop, laptops, tablet computers, smartphones, etc. Thesystem may also include motion sensors and other hardware that is morespecific to the functionality that will be described herein. In somecases, it will be appreciated that the computing system can beimplemented with such sophistication that it may be properly consideredthe “teacher” in some cases. And in this regard, it will be understoodthat many of the examples herein do not at all require the participationof a human teacher.

With respect to the example methods that will be described, the methodswill be described as including various steps, some of which will beshown in flowchart diagrams. These flowcharts may at first appear toimply that steps are performed serially in a particular order. And infact, it will often make sense that some steps are performed in aparticular sequence. That said, it should be understood that differentsequences may be employed, some steps may be performed concurrently withother steps, some steps may be omitted altogether, and still further,additional steps may be employed without departing from the spirit ofthe invention. Also, the example methods may be carried out using thehardware and software configurations herein, or using hardware andsoftware different from what is shown and described.

Continuing with the topic of steps, the present systems and methods maybe thought of in terms of sessions and cycles. For example, training astudent to perform a particular motion may include a first session thatmay variously be referred to as a communicating session, or ademonstration or instruction session. This often includes communicatinga model example of a bodily motion to the student. In some cases, thiswill include displaying a model video example of the motion. This firststage can also be thought of as an “assignment” session in the sensethat its purpose is often to assign an exercise or other motion to bepracticed.

After the demonstration/assignment stage, the training may include anabsorption session. In this session, the objective normally is to createan environment or conditions that increase the ability of the student tofocus upon and neurologically absorb the information they have receivedduring the prior session (e.g., during video display of a professionalathlete optimally performing an athletic motion). Absorption may includeplaying music or other audio, suppressing sensory inputs, facilitatingstates of relaxation, etc.

Finally, a practice session occurs, in which the student practices themotions. In some cases, it may be desirable to practice at differentspeeds, in order to facilitate and enhance the learning process.

Occasionally, a grouping or sequence of the above sessions may bereferred to as a training cycle, and a cycle can involve more than asimple three-session sequence such as demonstrating, then absorbing andthen practicing. For example, a cycle might include a first videodisplay session followed by an absorption session, followed by anotherdisplay session, and then another absorption session, and then threepractice sessions at different speeds. Any sequencing or grouping ofsessions can be thought of as a cycle. In some cases, a cycle ischaracterized by a particular focus. For example, in a first trainingcycle a golfer may be focusing on their overall golf swing. A secondtraining cycle for that athlete could focus more specifically on thegolfer's left arm. In some cases, a change in focus is made in responseto electronic observation of the student's performance, using motioncapture or other sensing technology as will be described in more detailbelow.

Referring back to and continuing with FIGS. 1 and 2, system 100typically will include a user interface 102, including a teacherinterface 102a and a student interface 102b. These interfaces allow theteacher and student to establish goals, specify particular motions topractice, and provide a variety of other inputs used to control andotherwise affect the training sessions and cycles. And the student andteacher may of course interact with each other in order to best takeadvantage of the features of the system and craft an effective course oftraining In addition to front-end manual inputs received through userinterface 102, a wide variety of programmatic inputs may influence andcontrol the training, meaning that such influence and control occurswithout the need for human intervention. In many cases, theseprogrammatic inputs are provided in very fast feedback loops, so as tobest tune the training to produce the best results. For example, if agolfer is more or less instantly informed of an error in some aspect oftheir golf swing, that will often provide the best opportunity toquickly and effectively make the necessary correction. Physical therapypatients receiving instant guidance will have better outcomes, and atlower cost given the leveraging of hardware and software technologiesthat can provide guidance with less time inputs from the therapist.

System 100 may further include a storage subsystem 104 for storing dataand software instructions to carry out the features of the system andmethod. Among other things, the stored content may include audio andvideo content that is presented to the student to help them learn thebodily motions of interest. For example, a video showing a professionalswimmer optimally performing a particular swim stroke may be stored instorage subsystem 104, for presentation to the student in one or moredisplay sessions. As will be described below, the student's practice ofthe motion may be recorded by video or other means, and this recordeddata may also be stored in storage subsystem 104. Audio content may alsobe stored, as will be described in further detail below. Beyond this,virtually any other type of data may be stored in storage subsystem 104.For example, the storage subsystem can store medical histories;information about injuries; information about past performances;information about settings used in particular training cycles and theresults obtained with those settings during physical practice; librariesof video and audio content; any of the inputs received via userinterface 102; etc. Any information relevant to motion training may bestored in storage subsystem 104 and used in various ways during thetraining And as indicated above, the storage subsystem containsexecutable instructions (e.g., instructions 105) to carry out the stepsof the methods described herein.

System 100 may also include a content creator/generator module 106; anoutput subsystem 108 including a display device and an audio outputdevice; absorption setup module 110; practice session configurationmodule 112; an electronic observation subsystem 113 including a sensor114 spaced from the student and/or wearable sensors 116 affixed to thestudent; and a processor 118. In keeping with the idea of assigningexercises to a student, module 106 may also be referred to as an“assigning” or “assignment” module or subsystem. In general, module 106generates and manages content which is output to the student by outputsubsystem 108; absorption setup module 110 configures and managesprovision of sensory-reduced sessions, described elsewhere herein, inwhich audio/video stimulus perceivable by the student is suppressed;practice configuration module 112 configures and manages practicesessions in which the student practices bodily motions; electronicobservation subsystem 113 observes and records information, e.g., aboutstudent's practice sessions. The processor can carry out any number offunctions, including the execution of instructions 105 for carrying outthe features, functions and method steps described herein.

As shown in FIG. 1, output subsystem 108 may also include a head-mounteddisplay 120, including a display device 122 (e.g., small screenspositioned in front of the student's eyes) and an audio output device inthe form of earphones 124. In addition to providing audio and videooutput, the display screens and earphones may be used to suppress one orboth of audio and visual stimulus perceivable by the student (e.g., byblackout out the screens, turning down volume, white noise, noisecancelling, etc.), so as to provide a sensory-reduced session.

Method 200 will now be described with occasional reference to thecomponents of system 100, though it will again be appreciated that otherhardware and software components may be employed other than those of theexample of FIG. 1. Briefly, and in general, method includes a globalsetup step 202 entailing a holistic, high-level design and configurationof the training; a setup step 204 for configuring the period in whichaudio/video instruction is provided to the student, e.g., using contentand assignment module 106; the providing of motion instruction to thestudent (206), for example with output subsystem 108 to deliveraudio/video; setup and conducting of an absorption period (208 and 210),for example using module 110 and head-mounted display or othersensory-reducing means to provide a sensory-reduced session; setup andcarrying out of the actual practice of the motion (212 and 214), e.g.,using practice configuration module 112 and providing audio/video outputwith output subsystem 108; and respectively at 216, 218 and 220:observation of the student's practice, analysis of the practice andother aspects of the training, and the use of feedback to influence andcontrol various aspects of the training

As indicated at 220, and which will be explained in detail below,feedback features may include, in response to and based on electronicobservation of student practice: (1) controlling, in any type ofsession, the use of audio, video, body stimulation, sensory reductionand/or training speed; (2) providing follow-up video content for viewingby the student, which differs in at least one aspect from previouslydisplayed content; (3) reflecting the existence, extent and nature of anobserved deviation in the student's practice from a desired performance;(4) emphasizing or providing indication of an observed deviation from adesired performance, where audio, video and/or body stimulation providesthe emphasis; and/or (5) real-time control of audio presented to thestudent.

Turning to global setup step 202, in this step the overall features of atraining cycle are established. Use of the term “cycle” again alludes tothe fact that the steps of method include stages performed in variousorders, and/or that are iterative and likely be performed repeatedlyduring the course of training a student. Global setup may includereceiving explicit inputs from the teacher and/or student, for examplethrough a user interface such as that shown at 102 (FIG. 1). In keepingwith the idea of a high-level global setup, this step may be anappropriate time to set longitudinal goals that are somewhat removedfrom the specifics of a particular bodily motion. A runner, for example,may want to achieve a top-10 placing in their age group, reduce apersonal best time by some amount, etc.

Again, a wide variety of inputs may be applied at the front end—overallgoals of a training program; medical history; information about pastperformances or past training regimens; and information about specificexercises that are to be practiced or performed. Athletes that performtimed events might include personal best times that have been achievedin the past; golfers might specify the distance they can achieve usingvarious golf clubs; athletes in general might include information aboutequipment they use. Selections might be made about particular videocontent or audio content to be presented to the student to help themconsciously and subconsciously develop a mental picture of how theirbody needs to move in order to achieve the desired progress. If anathlete would like to emulate the style of a particular professional,they could elect that all example video content be of that professional.For example, a swim student might elect to view performances from aparticular Olympic swimmer; a golfer might want to see a particularprofessional golfer; etc. These are but a very few of the nearlylimitless potential inputs.

In the context of inputs that influence the training, it should be againnoted that a wide variety of programmatically determined inputs may beemployed, in many cases as feedback received from other stages. Forexample, motion capture analysis may reveal that an athlete's practiceneeds improvement in one particular area. This information can then befed back and used to modify video content presented to the athlete in asubsequent video display. In particular, the new video content couldemphasize the particular aspect of the motion needing improvement.Another example of feedback is comparison of results obtained duringmultiple practice stages. The practice session in which the largestimprovement was achieved could be analyzed, for example, to determinewhat occurred in other steps leading up to the practice (e.g., how theabsorption was conducted). Such feedback could be used to make optimalselections for how the video presentation and absorption are to beconducted. In still another example, feedback can be used to control theplaying of audio content. Audio content is sometimes preferably syncedin a particular way to the student's practice. For example, it might beuseful to match the tempo of a song to the frequency of some repeatedmotion (e.g., a cyclist's pedal stroke). Motion capture could be used toassess the cyclist's actual cadence in real time, which in turn could beused to ensure that audio being played to the athlete was synchronizedwith the pedal strokes. In still another example, stimulation of anathlete's body might be tuned from a baseline regime based on a motioncapture determination that the athlete was having difficulty with someaspect of a motion. Electro-stimulation of a golfer's arm might beuseful for example, as a reminder to move the arm in a particular way.And such a need could be determined through a motion capture analysisthat this aspect of the golfer's motion was the issue that most neededto be addressed.

Yet another example of feedback could be a determination that an athleteexperienced the biggest improvement when listening to a particular songor other audio content. In such a case, that song could be automaticallyselected by software so that it would be played to the student at anappropriate time in a subsequent training cycle or stage. Perhaps atraining session requires a patient to perform a series of exercises,and video data or another observation method reveals that one or twoexercises in particular are not being performed to a satisfactory level,or that those exercises needed to be focused on for some other reason.Then the initial configuration at 202 could include making sure thatthose exercises were emphasized in an upcoming cycle. Feedback couldalso be used for motivational purposes, for example to positivelyreinforce that progress is being made, which in turn might cause thestudent to be more diligent or follow through with a course of trainingAlso, instead of negative feedback, electronic observation might revealthat mastery has been achieved for a particular exercise, in which casethe student would then be moved on to other exercises and new audio andvideo content geared toward that new motion.

It will be further understood that the feedback mechanisms herein arecapable of operating very rapidly, in order to provide feedback at apoint in time when it can be used to the greatest advantage. Forexample, in a conventional physical therapy setting, a patient visitsthe therapist's office and is guided through various exercises. Thepatient then leaves with instructions to perform various exercises athome. The patient can certainly self-observe how their home practiceprogresses, however that monitoring will be conducted without thebenefit of 3rd party objectivity, and even when a third party receivesinformation about the practice (e.g., at a subsequent office visit), thefeedback will be delayed in time by days or even weeks from when theactual home practice occurred. Also, it is quite possible that thepatient will not even recognize difficulties in the practice. Bycontrast, the present method can include automated video and motionsensing recording and immediate real-time analysis of the recorded data,which in turn can be leveraged more or less immediately to guide thepractice. Use of feedback can occur within seconds of observing thestudent's practice. And best practices can be uniformly adhered to, inthe sense that a fully researched model motion can be used as theyardstick which controls performance measurement and response to themeasured performance.

Continuing with FIG. 2, method 200 may further include, as shown at 204,a content creation/generation step, which may be performed, for example,by module 106 of FIG. 1. In most cases, video content will be desirablefor providing up-front guidance to the student as to the aspects of theideal motion. As previously described, the video content typically willinclude model video examples showing optimal performances of the bodilymotions to be trained. FIG. 3, for example shows a frame from a modelvideo example 300 of a golf swing. Such an example may be displayedusing the output subsystem of FIG. 1 (e.g., on a head-mounted displaysuch as shown in FIG. 1). Generally any type of example may be provided,for any movement or type of movement. A wide variety of other athleticmotions may be demonstrated by video (e.g., using renowned professionalathletes); physical therapy or other therapeutic movements may bedemonstrated; correct postures for sitting, lifting heavy objects, etc.may be shown; video of yoga poses can be displayed. The possibilitiesare limitless. And as previously discussed, virtually any type of input,whether manual or programmatic, whether feed-forward or part of afeedback mechanism, and from any other stage of the practice orcomponent of system 100 (when such hardware/software is employed), canbe used to select and generate audio and video content to be presentedto the student.

In addition to providing a whole unmodified example of a motion,modified or supplemental content may be provided or generated. Forexample, a video of a swimmer may include multiple versions in whichdifferent aspects of the swimming stroke are emphasized, for example thearms, legs, or the rotation of the torso occurring as the swimmer takesbreaths. Feedback may be used to select the appropriate versions. Forexample, assume that a model video example is displayed in a firstdisplay session. Feedback may then entail providing follow-up video in asubsequent display session, which differs in at least one aspect fromthe first-displayed example. For example, electronic observation of aswimmer might reveal that they were not kicking hard enough, and thefollow-up content could emphasize the legs shown in a model videoexample.

When emphasis or de-emphasis is employed in video, the non-emphasizedpart of the body may be rendered in black-and-white, with the emphasizedportion in color. As another example, the non-emphasized parts of thebody may be dimmed. Any method may be used to emphasize or de-emphasizeas necessary. Moreover, an entirely different video may be employed asfollow-up content, for example in the case where feedback or otherinputs dictate that the student move on to another movement or exercise.And it should be again emphasized that feedback inputs can be positive.New video content might be selected after a student has mastered anexercise, the new video content being, for example, a more difficultexercise that the student has demonstrated they are ready for.

Video content, whether in an initial display session or a subsequentdisplay, can have various other characteristics. Various objects in themay be occluded, for example. In follow-up feedback video, occlusion maybe used to emphasize or de-emphasize certain elements, for example inresponse to observed deviations from a desired performance. Luminescenceand color variations may be employed for various purposes, including tohighlight observed problems with the practice. Follow-up video contentmay be edited to only show particular aspects that were shown in a priordisplay, again to emphasize deviations or for other purposes.

FIGS. 4, 5 and 6 provide illustration of feedback and how step 204 canbe influenced by that feedback. Let's first suppose that a golfingstudent was first shown model video example 302 in a first viewingsession (i.e., iteration of step 206). Then, during practice 214, motioncapture video is employed (step 216) to observe the practice. Thefeedback provided to the student may occur in a subsequent iteration ofstep 206. In this second session, the student may be shown stills orvideo of their actual performance, as shown at 400 in FIG. 4. FIG. 5provides another example of the feedback-generated content that can beshown to the golfer, in the form of an overlay of their performance onto the model performance—overlay video shown at 500 in FIG. 5. A note onFIGS. 3-6: while the clothing, body shape, etc. appears the same for thefigure in bold lines as the one in dashed lines, this replication issimply for purposes of simplicity, particularly with respect to theoverlay. In actuality, the actual student will appear in the stills andvideo, and they will of course look very different than the professionalathlete appearing in the model video example.

After the student has seen their actual performance, a modified versionof the model example can be shown, as follow-up video content in asubsequent display session, in which a particular aspect of the golfswing is emphasized. FIG. 6 shows an example of such follow-up content.Here the video example 600 is similar to example 300 of FIG. 3, exceptthat the arms are emphasized, because it was determined in theobservation 216 and analysis 218 steps that the golfer's main difficultywas their arm positioning. Electronic observation thus has beenefficiently leveraged to tune the training to emphasize the specificarea in which the student needs to improve.

Content that is presented to the student at step 206 (and other steps)can also include audio. Any type of audio may be employed, although theinventors have determined that some types of audio provide specific andclear advantages in certain settings. The audio may include astudent-selected song. A song or other audio clip could be selectedbased on it having a particular tempo, which can be advantageous intraining repetitive motions that the student should perform at aparticular frequency. In this regard, a 400-meter runner might select ahigh-energy motivational song having a tempo that matches their optimalstride for the 400-meter distance. As another example, cyclists oftenfocus closely on pedaling cadence, such that selection of music with aspecific tempo can be quite helpful. Another type of audio content isbinaural beats.

At step 206, method 200 includes the actual presentation of content tothe student. As indicated above, this step may be variously referred toas “instructing,” “demonstrating,” “presentation,” or “assigning,” whichreflects that distinctions are appropriate in some cases based on theprecise purpose to be achieved. For example, in many cases, the term“demonstration” is clearly applicable; many examples include using avideo to demonstrate the motion to be performed. “Assigning” can referto step 206 being carried out to assign particular motions or aspects ofmotions to practice. A teacher in the form of a physical therapist can“assign” specific exercises for a patient, and can also “instruct” thepatient so as to enhance and improve the practice experience. And theword “presentation” will also be appropriate, for example the playing ofaudio to the student may be naturally described as a “presenting”activity. However, setting aside these subtleties, the essence of step206 is that the student is given information about the motion that theyare to practice. The providing of information may include displayingvideo and/or playing audio, as has already been discussed. In otherexamples, body stimulation may be employed during step 206. In theexample of FIG. 1, content is provided to the student via the outputsubsystem, which may include one or more display devices and audiooutput devices.

If audio is employed at step 206, it may be useful to re-play that sameaudio (or with certain modifications) during absorption 210 and practice214 sessions. Although the audio may be played in various ways,including with modifications, additions and/or deletions, it will oftenbe helpful to play it in a way substantially similar to when it wasplayed during step 206. Importantly to some scenarios, the audio will besynchronized to link aspects of the motion shown at 206 (when video isemployed) with the student's attempts to perform those aspects—e.g., thesame moment in the audio occurring as the model swimmer places theirright arm in the water would be played as the student is placing theirright arm into the water during the practice. In some cases, this may bereferred to as “maintaining an audio-motion synchronization.” In otherwords, each moment of the audio has a corresponding associated moment inthe motion, whether the motion is displayed in a video or is beingpracticed by the student. When such synchronization is employed betweenthe demonstration and practice, the synchronized audio during practicecan aid in properly activating the neuromuscular systems needed toproperly perform the motion. Common, synchronized audio can provide apowerful “neurological anchor” that beneficially links thedemonstration, absorption and practicing of the motion.

Typically it will be desirable that the audio content and its deliverybe configured to induce the student to cognitively absorb the modelvideo example, particularly during sensory-reduced sessions. This mayinclude selection of a tempo, for example so that audio events aresynchronized with particular motion events. This may be particularlyuseful for repetitive motions occurring periodically. In addition,patterned and directed audio may be dynamically time warped to replicatea programmed motor pathway that meets predetermined speeds or affordsadjustable speeds to induce cognitive learning for skill acquisition tobecome task oriented.

Regardless of whether played during demonstration, absorption orpractice, the control of audio may be based on a variety of inputs.Feedback inputs based on electronic observation of practice can beparticularly useful. The feedback in general may include controllingplaying of audio during one or more of: (i) a subsequent displaysession; (ii) a subsequent sensory-reduced session; and (iii) the firstpractice session; and (iv) a subsequent practice session.

During subsequent displays of video content, audio may be controlled toprovide or emphasize an area that needs improvement, which may have beendetected by electronic observation. Such areas can also be highlightedwith video content, using methods of emphasis or de-emphasis asdescribed above. During sensory-reduced sessions, audio control based onfeedback can include: speed control (e.g., in response to determiningusing motion capture that display/absorption/practice should beadjusted, for example made slower, to facilitate learning); and volumecontrol for various purposes, including emphasis of particular aspectsof the motion being practiced. Another example of control during asensory-reduced session is an observation that a particularly-configuredsensory-reduced session was followed by a particularly good practicesession. In such a case, control could be performed to repeat the sameconfiguration in subsequent sensory-reduced sessions.

During the practice itself, motion capture can ensure audio is played atan appropriate tempo and properly synchronized to the motions thestudent is attempting to perform. When the student deviates from adesired performance, immediate audio feedback can be provided, forexample as a volume change, distortion, beat tones or other warning-typesounds, removal or addition of other audio components, etc. Theelectronic observation system may also observe the student practicing atother than a desired speed, in which case audio playback can becontrolled to prompt or induce the student to speed up or slow down thepractice. And again, the feedback generally will be performed based onobservation 216, analysis 218 (e.g., is the deviation large enough to bedetermined an insufficiency) and feedback 220.

As indicated above, method 200 also includes a period in which thestudent focuses on the video or other information provided at 206, inorder to consciously and subconsciously absorb and internalize themotion being learned. At step 208, the absorption environment isconfigured to provide such a sensory-reduced session, which can includeselection of audio and or video content to be played, including inmodified forms. Selections may also be made at this step regardingsuppressing the ability of the student to take in sensory inputs fromtheir environment, so as to produce a sensory-reduced session. A devicesuch as head-mounted display 120 may be used to black out a display,provide “white noise” audio or video, noise cancelling/blocking, etc.Such a sensory suppression can enhance absorption in some cases.Usually, one goal of the absorption period is to design it so as toinduce a state of relaxation, so as to place the student in an emotionaland mental state in which they are receptive so as to optimally absorbthe content. Suppressing audio and/or visual stimulus can facilitaterelaxation, eliminate distractions, and otherwise enhance focus,visualization and internalization of the information. And as before, theparticular settings may be influenced or determined by virtually anytype of input, programmatic or manual, feed forward or feedback, andfrom any step or software/hardware component. In one example, ananalysis may be performed as to what absorption settings were followedby the best performances of the student, so that the most helpfulsettings can be replicated in upcoming absorption (e.g., via steps 216,218 and 220). The absorption period itself occurs as shown at 210.

Audio content may have other characteristics to enhance training Audiocontent may be mapped to velocities and positions for varioussignificant mechanical points of interest in a bodily motion. Audio maybe structured to have specific notes, pitch, frequencies, binauralbeats, etc., regardless of whether played during video display, sensorysuppression or practice. During sensory reduction, the notes, pitch,binaural beats, etc. are structured to maximize attentiveness andrelaxation. The audio content employed may be selected based onempirical determinations of its ability to achieve cognitive andphysical benefits. Also, as discussed elsewhere herein, studentperformance over multiple cycles and sessions can be evaluated in orderto identify optimal characteristics and uses of audio content.

Steps 212 and 214 pertain to the actual practice of the movement by thestudent. At 212, the upcoming practice session is “configured,” in thesense that selections are made with respect to the particular movementor aspects of movement to be practiced; the environment in which themovement will be practiced; whether and what type of audio and videowill be provided to the student during the practice; whether and how thepractice will be observed and monitored; whether and how the studentwill be provided with real-time feedback as the practice occurs; thespeed at which the motions will be practiced; whether haptic or othersensory stimuli will be provided, etc. As with other setup steps, theseselections may be made with any type of inputs, as discussed above andincluding feedback based on observation 216 and analysis 218. The actualpractice is carried out at step 214.

Practice may be carried out at various speeds. A given speed may bedetermined in advanced and enforced via various methods. Enforcement ofspeed may include controlling an audio track to play at a particulartempo, or controlling the frequency of a repeated tone or sequence oftones. Speed control can be used to prompt changes in the student'sspeed. For example, if the student is moving slower than desired, thetempo of accompanying music can be increased slightly, so as to inducethe student to “chase” the audio and converge to synchronicity. On theother hand, audio can be slightly slowed to signal a need for decreaseand induce such decrease. Enforcing practice speeds can be veryhelpful—one can well imagine that an optimum path to mastery would be tofirst start at a slower pace. And speed control could rein in theovereager student who wants to proceed at a fast tempo before they havea sufficient grasp of the basics of the movement.

On the other hand, it may be useful to let the student proceed duringactual practice at a speed that is comfortable for them. In this case,which involves a non-predetermined speed that may fluctuate, it will beuseful to implement certain controls based on a real-time observation ofthe practice. Again, this would involve the previously-referenced steps216, 218 and 220, with motion capture or another mechanism forrecognizing the speed of the practice. Knowing the speed may beimportant if it is desired to play audio or otherwise provide thestudent with stimulation or other information at specific times, forexample wanting an audio segment to occur while the student ispracticing a particular aspect of the motion. Whether speed is enforcedon a predetermined basis or allowed to occur organically, any range ofspeed may be employed. Indeed, speed may range from very slow to a ratebeyond anything that would be desirable during actual practice of theactivity (e.g., faster than a particular dance step would ever actuallybe performed in a normal dance setting).

As previously indicated, the efficacy of the systems and methodsdescribed herein can be greatly enhanced via observation and monitoringof the student's performance. Indeed, example method 200 includes, at216, electronically observing the student's practice. Electronicobservation may be performed, for example with electronic observationsystem 113, using optical technologies such as time of flight,structured light, marker tracking with active or passive markers, andnon-optical methods, such as with accelerometers, gyroscopes, magnetictracking, etc. Other data may also be obtained, such as heart rate,respiration rate, work rate (e.g., strokes/strides/revolutions perminute), time needed to perform an exercise or cover a specifieddistance, etc.). At 218, analysis may be performed, which in turn canproduce feedback that can send inputs to or control other stages, asshown at 220. The possibilities for control based upon observation andanalysis are limitless.

As mentioned above, observation at step 216 is often conducted todetermine that the student's practice of the motion has deviated fromsome desired performance (e.g., arm at the wrong angle, incorrectposture, timing of a motion being early/late, etc.). Feedback canreflect this deviation, and in some cases will vary with the extent ofthe deviation. The deviation can affect the setup, configuration andoutput provided at steps 202, 204 and 206. As an initial matter, thestudent may be shown video or other information recorded about theirpractice (see FIG. 4, a video of student's actual performance; and FIG.5, a video overlaying the actual performance on to of the model exampleto facilitate recognition of problem areas). Continuing with the case ofrecognized deviations, analysis might reveal that the deviationindicates that the current bodily motion is too difficult for thestudent, which in turn can lead at 202 to the selection of a moreappropriate bodily motion to practice. On the other hand, analysis ofthe deviation might result in a configuring, at step 202, so that thesubsequent steps continue with the same motion, but with an emphasisgeared toward correcting the specific deviation. Analysis might revealthat the student's performance has characteristics that carry anincreased risk of injury, and this information may be used in steps 202and 204 to modify subsequent stages to address the risk. For example, ifa weightlifter was using a risky hand position, modified video contentemphasizing correct hand placement could be selected at 204, forpresentation to the athlete in an upcoming demonstration stage 206 (seeFIG. 6).

Another example: an observed deviation might affect or control the audioplayed to the student during stage 206. Suppose that multiple iterationsof step 206 had occurred, and that the subsequent practice at iterationsof step 214 had produced varying results. The analysis could identifythe audio that was played at 206 that lead to the best performance(e.g., with the least deviation). This audio would then be selected forsubsequent iterations of the 206 step. In another example, audio can becontrolled to provide cues that occur at specific times as the studentpractices. If the observed deviation occurred for only a few momentsduring the student's practice, the audio in 206 could be varied at thecorresponding moment when a model video example is being played.Specifically, if a golfer's deviation occurred at follow through, thefeedback control of audio can include, while the golfer is subsequentlywatching a video model example, changing the audio at the moment offollow through, to emphasize that aspect of the motion.

In some cases, it will also be desirable to play audio content duringthe absorption stage 210. Control of this audio based on an observeddeviation can also be performed. Referring to the above example whereaudio is varied at the moment of deviation, a similar feedback-basedtiming may be used. Another example of audio control duringdemonstration and absorption would be a determination that a differentspeed should be used. For example, if the student's performance atslow-speed practice has improved significantly, then the video andassociated audio played in steps 206 and 210 could be played faster.

Deviation-based feedback control can also affect the actual practice. Asjust mentioned, electronic observation of a deviation can be used tovary practice speed or the speed used on other stages. For example, if alot of deviation were observed, then the feedback may result in practicestage 214 being conducted at a slower speed. Real-time audio feedbackcan be used as well. For example, at the moment of deviation, an audiovariation can be introduced to alert the student to make a correction.Audio variation can be introduction of new content, change in volume,introduction of distortion, to name a few non-limiting examples.Electronic observation may indicate practice occurring at other than adesired speed, and audio can be controlled to prompt the student toincrease or decrease the speed of the practice.

As mentioned above, varying the speed of practice can enhance trainingIn addition, it will often be useful to control speed of audio and videoduring display sessions (e.g., at step 206 in FIG. 2) andsensory-reduced sessions. In many cases it will be helpful to begin at arelatively slow speed, with video demonstration of bodily motion andassociated audio content beginning at a relatively slow speed. Slowspeeds allow the student to focus more closely on a motion or particularaspects of the motion. In particular, slowing down video to below-actualspeeds can greatly enhance learning. Typically, the same audio contentwould then be played at the same speed in a sensory-reduced sessionfollowing the display. The use of the same audio can help to inducegreater cognitive absorption of the model video example shown to thestudent. Repeated sounds during the absorption period can focus thestudent on the aspects of the motion that coincide with those sounds.And in some cases, this benefit will be greater if the same speed isused. That said, a different speed may be used in the display andsensory-reduced sessions.

Training typically includes multiple iterations of the differentsessions (display, absorption, practice), and speed will often bechanged in subsequent sessions. For example, video and audio speed willoften be increased in subsequent display and absorption sessions. Thesesubsequent sessions can occur during a given workout, or they may occurdays or weeks later. Generally, a speed increase is used as the studentimproves. It will also be understood that video and audio speeds may bedecreased, for example if the student's performance deteriorates.

Audio/video speed control will often be implemented as feedback based onelectronic observation of performance. Improved performances can lead toincreased speed in subsequent display and absorption sessions.Similarly, where a student's performance shows some difficulty ordeviation from a desired performance, subsequent sessions can be sloweddown. These are but examples, there may be other reasons to change speedbased on performance.

Returning to control of video, in general, and to summarize, thefollowing video control may be provided as followup video content inresponse to and based on electronic observation of the practice: (1)speed variations; (2) visual emphasis of certain aspects; (3) occlusion;(4) controlling luminescence; (5) selection of different video content;(6) editing prior video content to retain and display only a portion ofthe prior content; (7) color modifications; etc. One, some or all ofthese controls may be used more specifically in response to observeddeviations from a desired performance. These may also be based onimprovements or other electronic observations of the practice. Again,for example, analysis might reveal a student has achieved mastery in aparticular exercise, in which case the follow-up video could providepraise or some other indication of success, and/or video of a new motionor motion aspect to be trained. Performance may indicate a speed changewould be helpful, even though there is not a specific problem area.

Regarding audio, it has been discussed that audio may be configured toinduce enhanced cognitive absorption of the motion being trained. Inaddition, and in summary, audio may be controlled based on electronicobservation: (1) to adjust speed, for example based on observing thatsubsequent sessions should be performed at a different speed, or toaccount for observed difficulty, mastery or improvement; (2) to adjustspeed to synchronize audio to actual observed performance; and (3) tocontrol volume, distortion, adding/removing components, etc., inresponse to observed difficulty, mastery, improvement, etc. These arenon-limiting examples—a variety of other audio controls may be employedin different settings.

Still further, with respect to audio and/or video, electronicobservation can be used to compare performances occurring in differentpractice settings. The systems and methods herein may be tuned byanalyzing what audio and video settings, changes, etc. were followed bythe best performances. These settings can then be replicated insubsequent training activities.

Referring now to FIGS. 7 and 8, FIG. 7 shows an example method 700 fortraining bodily motions using body stimulations, and FIG. 8 showsaspects of the method in reference to depicted swimmer 800.Incidentally, FIG. 8 provides another example of video content that canshow an actual performance deviating from a desired performance—here oneof the swimmer's legs (dashed) is lower in the water than in the modelexample (in solid lines).

Continuing the topic of stimulations, is In some implementations,stimulating the student's bodies in selected locations can enhancemotion training Stimulation may be performed using various methods andtechnologies. In some examples, electrical stimulation is performed withelectrodes. In other examples, pressure, vibration, temperature, touchor other haptic signals and stimulation are used. As seen in FIG. 8, anumber of stimulators 802 are affixed to the body of swimmer 800. Thesestimulators may be electrodes adapted to stimulate muscular activity,haptic devices that provide pressure, compression, vibration, heat,cold, etc. In some implementations, these devices may be incorporatedinto a garment worn by the student, such as a swimsuit. The stimulatorstypically are driven by signals (e.g., wireless signals) that are outputas a result of executing software instructions, for example execution ofinstructions 105 with processor 118.

Referring back to the previous discussion of motion capture, theplacement of stimulators 802 can be used for marker-based motioncapture. Alternatively, those locations can correspond to trackedlocations in marker-less motion capture methods. Still further thelocations may correspond to motion/position-sensing devices such asaccelerometers.

Continuing with body stimulation, regardless of the particularstimulation method, it typically will entail, for a given bodily motion,decomposing the motion into a plurality of motion events that occur whenthe motion is properly executed. The definition and selection of motionevents is shown at 702 in method 700. Each event is associated with andinvolves a particular body part. For example, the follow-through in agolf swing involves a driving motion (motion event) of the hips (theassociated body part). A given motion typically will have several motionevents, e.g., bringing the golf club backward from an initial restingposition; a middle portion of the backswing; the full retraction of thebackswing and the attendant position of the arms; the beginning of theforward swing; the position of the head during the forward swing; thedriving of the hips during follow through, etc.

The stimulation method may then include, as a model video example of themotion is played to the student, and for each motion event, stimulatinga location on the student's body that corresponds to the motion eventand its associated body part. Also, the stimulation is timed to occur atthe same time that the motion event is shown in the displayed videoexample. This is shown at steps 704 and 706.

In addition to applying stimulation during the display of the video,stimulation can also be applied during absorption and/or the actualpractice of the motion. When employed during a sensory-reduced session,it will often be useful to employ the same timing of stimulation as wasemployed during video display of the bodily motion, so as to strengthenthe mental/physical link between the two stages and enhance theabsorption. Stimulation during a sensory-reduced session is shown at706.

As shown at 710, stimulations may also be applied during practice of thebodily motion. Typically, stimulations are applied for each motion eventand associated body part so that the stimulations are synchronized withthe student's attempts to perform the motion events. Thissynchronization may be performed using electronic observation, asdiscussed above, in order to determine when the student is attempting toperform the motion events. During practice of the motion, it will oftenbe useful to again use the same timing. That said, the student mayintentionally or unintentionally practice at a different rate than thatshown in the video example. The different rate could be specified andcontrolled somehow, or could just naturally result from the way thestudent practiced at a given instant. In such a case, electronicobservation (e.g., via machine vision motion capture) can again be usedto control the timing of stimulation during the practice, so that thestimulation occurs at an appropriate location on the student's body atthe time that the student is attempting to perform the motion event.

Step 712 shows controlling of the stimulations. This control can includestimulation control during video display sessions, sensory-reducedsessions, and/or practice sessions. The control can include speedcontrol; magnitude of stimulation; control based on electronicobservation of practice; control to emphasize certain motion eventsrelative to others; and/or control based on an observed deviation from adesired performance.

Regarding speed control, different stimulation speeds and timings may beemployed. This can include an overall uniform speed change across thewhole bodily motion. Alternatively, speed may be slowed or increasedonly for portions of the bodily motion (e.g., for a subset of the motionevents). In some cases, training cycles may be arranged to have apre-defined use of different speeds. For example, a slow speed might beused for initial display, sensory-reduced and/or practice sessions, withspeed being varied in subsequent sessions, for example speeding up aspractice improves. Selecting a speed may be performed based onelectronic observation of practice, for example to emphasize a difficultarea (e.g., to reflect an observed deviation on a particular motionevent), or based on an observation that the student would benefitsomehow from a different speed (e.g., based on mastery at a slow speed).

As just briefly mentioned, electronic observation (e.g., with system 113of FIG. 1), can be used to control stimulation. This can be used tocontrol any types of session (display, sensory-reduction, practice). Ina current practice session, speed can be changed, stimulations can beapplied to emphasize certain motion events, stimulations can signal whenthe student is deviating from a desired performance, etc. In asubsequent display, sensory-reduction or practice session, certainmotion events can be emphasized; stimulation speeds can be changed; etc.In particular observed data might be analyzed to conclude thatadditional video/absorption/practice should be conducted at a differentpace, such as to slow practice down in the event of difficulties.

In general, and to summarize with respect to feedback based onobservation, feedback-based control can affect any characteristic ofstimulation, both in a current practice session and in any subsequenttype of session. The control can include (1) controlling stimulationspeed, e.g., to increase or decrease speed in subsequent sessions, forall motion events or any subset of those events; (2) magnitude ofstimulation; (3) emphasizing certain motion events, including deviationsfrom desired performance; (4) changing the subset of events for whichstimulation is performed; and (5) activating additional stimulationsites; etc. These are but examples, any practicable control can beperformed in response to electronic observation. And as with audio andvideo, electronic observation can be used to tune optimal stimulationsettings, for example by analyzing what stimulation methods have causedthe greatest improvements in performance. In addition, any of the abovemethods relating to providing and/or controlling video and audio indisplay, absorption and/or practice sessions may be combined with bodystimulation to aid in the training of bodily motions.

It is to be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated may beperformed in the sequence illustrated, in other sequences, in parallel,or in some cases omitted. Likewise, the order of the above-describedprocesses may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A method for training a student to perform a bodily motion,comprising: communicating a model example of the bodily motion to thestudent; subsequent to the communicating, and while the student ispracticing the bodily motion in a first practice session, electronicallyobserving the student's practice of the bodily motion with an electronicobservation system; and providing feedback to the student based on theelectronic observation.
 2. The method of claim 1, where thecommunicating includes displaying a model video example to the studentin a first display session.
 3. The method of claim 2, furthercomprising: determining, as a result of the electronically observing,that the student's practice of the bodily motion in the first practicesession deviates from a desired performance and where the feedbackreflects such deviation.
 4. The method of claim 3, where feedbackemphasizing the deviation is provided in video content in a seconddisplay session.
 5. The method of claim 3, where the feedback includesproviding audio content to the student that emphasizes the deviation. 6.The method of claim 2, further comprising: playing audio content to thestudent during the first display session; and playing the audio contentagain during the first practice session, and where the feedback includesdynamically controlling the playing of the audio content during thefirst practice session so as to maintain an audio-motion synchronizationwhich is the same as that employed in the first display session.
 7. Themethod of claim 6, where providing the feedback includes, during thefirst practice session, providing an audio indication when the student'sperformance deviates from a desired performance.
 8. The method of claim2, further comprising: subsequent to the first practice session,displaying follow-up video content to the student, such follow-up videocontent being different in at least one aspect from the model videoexample and generated in response to and based on electronic observationof the first practice session.
 9. The method of claim 8, where theelectronically observing includes observing that the student'sperformance in the first practice session deviates from a desiredperformance, and where the follow-up video content emphasizes thedeviation.
 10. The method of claim 2, further comprising, between thefirst display session and the first practice session, suppressing one orboth of audio and visual stimulus perceivable by the student to therebyprovide a sensory-reduced session.
 11. The method of claim 10, furthercomprising: playing audio content to the student during the firstdisplay session; and playing the audio content to the student during thesensory-reduced session.
 12. The method of claim 11, further comprisingplaying the audio content to the student during the first practicesession.
 13. The method of claim 2, where the electronic observationsystem includes an optical motion capture system.
 14. The method ofclaim 2, where the electronic observation system includes wearablesensors affixed to the student.
 15. The method of claim 2, where audiocontent and video content is provided to the student using ahead-mounted display which includes an audio output device, and wherethe head-mounted display is configured to suppress one or both of audioand visual stimulus perceivable by the student to thereby provide asensory-reduced session.
 16. The method of claim 2, where providing thefeedback includes playing the model video example in a second displaysession at a speed different than that used in the first displaysession.
 17. The method of claim 2, further comprising: playing audiocontent associated with the model video example during the first displaysession; suppressing one or both of audio and visual stimulusperceivable by the student to thereby provide a first sensory-reducedsession; and playing the audio content during the first sensory-reducedsession, where the providing the feedback includes again playing theaudio content in one or both of a subsequent display session and asubsequent sensory-reduced session, but at a different speed than usedduring the first display session.
 18. A system for training a student toperform a bodily motion, comprising: an output subsystem, storagesubsystem, electronic observation subsystem and feedback subsystemoperatively coupled together, the storage subsystem containingexecutable instructions operative to: cause the output subsystem todisplay to the student a model video example of the bodily motion in afirst display session; cause the electronic observation subsystem to,after the first display session, electronically observe the studentpracticing the bodily motion in a first practice session; and cause thefeedback subsystem to generate and provide feedback to the student basedon the electronic observation of the first practice session.
 19. Thesystem of claim 18, where the electronic observation subsystem isoperative to determine that the student's practice in the first practicesession deviates from a desired performance, and where the feedbackreflects such deviation.
 20. The system of claim 19, where the feedbackincludes playing video content that emphasizes the deviation.
 21. Thesystem of claim 19, where the feedback includes playing audio content tothe student that emphasizes the deviation.
 22. The system of claim 18,where the output subsystem is operative to: play audio content duringthe first display session; and play the audio content during the firstpractice session, where the feedback includes controlling the playing ofthe audio content during the first practice session so as to maintain anaudio-motion synchronization which is the same as that employed in thefirst display session.
 23. The system of claim 22, where providing thefeedback includes, during the first practice session, providing an audioindication when the student's performance deviates from a desiredperformance.
 24. The system of claim 18, where the feedback includesdisplay of follow-up video content to the student in a second displaysession, such follow-up video content being different in at least oneaspect from the model video example and generated in response to andbased on electronic observation of the first practice session.
 25. Thesystem of claim 24, where the electronic observation subsystem isoperative to determine that the student's practice of the bodily motionin the first practice session deviates from a desired performance, andwhere the follow-up video content emphasizes such deviation.
 26. Thesystem of claim 18, where the electronic observation subsystem includesan optical motion capture system.
 27. The system of claim 18, where theelectronic observation subsystem includes wearable sensors affixed tothe student's body.
 28. The system of claim 18, where the outputsubsystem includes a head-mounted display with an audio output device,and where video and audio content provided to the student is performedusing the head-mounted display, and where the head-mounted display isalso configured to suppress one or both of audio and visual stimulusperceivable by the student to thereby provide a sensory-reduced session.29. The system of claim 18, where providing the feedback includesplaying the model video example in a second display session at a speeddifferent than that used in the first display session.
 30. The system ofclaim 18, where the executable instructions are further operative to:cause the output subsystem to play audio content associated with themodel video example in the first display session; cause the outputsubsystem to play the audio content during a first sensory-reducedsession, in which one or both of audio and visual stimulus perceivableby the student are suppressed, where the feedback includes again playingthe audio content in one or both of a subsequent display session and asubsequent sensory-reduced session, but at a different speed than usedduring the first display session.
 31. A method for training a student toperform a bodily motion, comprising: displaying a model video example ofthe bodily motion to the student in a first display session; playingaudio content during the first display session; subsequent to the firstdisplay session, suppressing sensory stimulus perceivable by the studentto thereby provide a first sensory-reduced session; playing the audiocontent during the first sensory-reduced session; subsequent to thesensory-reduced session, and while the student is practicing the bodilymotion in a first practice session, electronically observing thestudent's practice of the bodily motion with an electronic observationsystem; controlling, based on the electronic observation, subsequentdisplay of video content to the student in a second display session; andcontrolling, based on the electronic observation, subsequent playing ofaudio content in one or both of a second display session and a secondsensory-reduced session.
 32. The method of claim 31, further comprisingcontrolling, while the student is practicing the bodily motion, theplaying of audio content during such practice, the audio content beingcontrolled based on electronically observing the practice as it isoccurring, so as to maintain an audio-motion synchronization that is thesame as that employed while the audio content was being played during adisplay session preceding the practice.
 33. The method of claim 31,further comprising determining, based on the electronic observation ofthe first practice session, that the student's performance in thatsession deviated from a desired performance, and where subsequentcontrol of audio or video presented to the student reflects thedeviation.